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Abstract —  Developing  cyber  engineering  solutions  for  the  Defense 
Department  requires  decisions  that  affect  the  cost,  schedule,  and 
performance  of  not  only  the  constituent  system  but  those  of  the 
combined  end-to-end  System  of  Systems.  Considerable  research 
has  been  conducted  on  the  topic  of  decision  aiding  methods  such 
as  Multi-criteria  and  Multi-objective  Decision  Analysis  to  support 
results  given  the  uncertainties  within  the  acquisition  environment. 
Besides  the  problem  definition  itself,  the  most  significant 
contribution  to  a  decision  model’s  success  is  the  identification  of 
the  correct  key  decision  criteria  to  meet  the  stakeholder’s  goals. 
Unfortunately  not  all  of  the  decision  makers  will  agree  on  what  is 
most  important.  In  essence,  the  system  engineer’s  choices  and 
weighting  may  be  significantly  different  from  those  of  the  program 
manager,  resource  sponsor,  or  even  the  user.  This  research  focuses 
on  the  use  of  recursive  sensitivity  analysis  to  mitigate  the 
uncertainty  that  may  be  introduced  through  the  bias  of  the  Subject 
Matter  Experts  queried  for  the  Multi-Criteria  Decision  Modeling. 
The  application  of  sensitivity  analysis  to  the  criteria  selection  and 
weighting  process  prior  to  and  directly  following  the  decision 
aiding  methods  could  significantly  reduce  ambiguity  and 
ultimately  improve  the  quality  of  the  decision. 

Index  Terms —  Decision  making,  defense  acquisition,  operations 
research,  optimization,  sensitivity  analysis,  systems  engineering. 

I.  Introduction 

HE  selection  between  technical  alternatives  to  fill 
capability  gaps  identified  for  our  warfighters  is  arguably 
the  most  significant  decision  in  defense  acquisition.  This 
critical  decision  directly  affects  the  cost  of  the  program,  the 
time  necessary  for  development  and  delivery,  as  well  as  the 
ability  to  leverage  technology  to  increase  the  functionality 
provided  to  the  user.  In  light  of  the  numerous  cyber-attacks  of 
late,  many  in  the  Department  of  Defense  (DoD)  are  seeking  to 
increase  the  speed  at  which  cyber  capability  can  be  acquired 
and  integrated  into  the  force.  The  initial  selection  of  a  solution 
concept  or  an  engineering  alternative  significantly  impacts  this 
acceleration.  Georgiadis,  Mazzuchi  and  Sarkani  estimated  that 
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the  requirements  determination  and  technical  alternative 
selection  effort  is  only  2%  of  the  total  life  cycle,  but  this 
formative  decision  drives  the  remaining  98%  of  the  acquisition 
efforts  [1].  Therefore,  it  is  essential  that  research  focuses  on 
modeling  decision  criteria  weighting  and  equilibria  points  to 
ensure  the  optimal  alternative  is  selected. 

A.  Analysis  of  Alternatives 

For  most  major  defense  acquisitions,  significant  investments 
in  time  and  resources  are  applied  to  evaluate  each  of  the  feasible 
alternatives  with  respect  to  criteria  such  as  life-cycle  cost, 
technical  maturity,  system  performance,  and  schedule.  This  is 
directed  by  both  statutory  and  regulatory  policy  in  the  Weapons 
Systems  Acquisition  Reform  Act  (WSARA,  Public  Law  111- 
23)  [2]  and  the  Department  of  Defense  Instruction  on  the 
Operation  of  Defense  Acquisition  System  (DoDI  5000.2)  [3]. 
This  evaluation,  known  as  an  Analysis  of  Alternatives  or 
“AO A”,  is  an  attempt  to  objectively  offer  the  Milestone 
Decision  Authority  (MDA)  the  best  alternative  to  meet  the 
validated  requirement.  However,  the  U.S  Government 
Accountability  Office  (GAO)  reported  that  “AO As  have  often 
simply  validated  a  concept  selected  by  the  sponsor  and  are  not 
used  as  intended  to  make  trade-offs  among  performance,  cost 
and  risks  to  achieve  an  optimal  weapon  system  concept  that 
satisfies  the  warfighter’s  needs”  [4]. 

This  demonstrates  a  distinct  possibility  of  the  introduction  of 
subjectivity  and  bias  into  the  AO  A  results  based  on  the  criteria 
and  importance  weighting  of  the  Subject  Matter  Experts  (SME) 
conducting  the  evaluation.  The  GAO  also  found  that  “Most  of 
the  programs  reviewed  either  did  not  conduct  an  AOA  or 
conducted  an  AOA  that  focused  on  a  narrow  scope  of 
alternatives  and  did  not  adequately  assess  and  compare 
technical  and  other  risks  of  each  alternative” [5].  In  2012,  the 
Senate  Armed  Services  Committee’s  characterized  this 
subjectivity  as  “cultural  bias  [that]  produces  overly  optimistic 
cost  and  schedule  estimates  and  unrealistic  performance 
expectations”  [6]. 
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There  is  an  abundance  of  literature  recommending  the  use  of 
operational  research  methods  or  decision  analysis  tools  to  assist 
in  the  conduct  of  an  AO  A  [7],  [8],  [9].  Most  of  these  advocate 
the  use  of  Multi-Criteria  Decision  Making  (MCDM)  models 
that  can  compare  multiple  attributes  of  a  discrete  set  of 
alternatives  offering  the  best  ranking  alternative  based  on  the 
preference  values.  The  intent  of  this  research  is  to  offer  an 
evaluation  of  techniques  that  may  reduce  the  bias  of  the 
preference  values  entered  into  these  MCDM  models.  The 
assumption  is  that  through  the  recursive  or  iterative  use  of 
sensitivity  analysis  techniques,  overly  influential  criteria  or 
mutually  cancelling  criteria  from  different  experts  may  be 
balanced  to  optimize  the  recommended  solution. 

II.  Multi-criteria  decision  making  tools 
There  are  many  MCDM  methods  and  supporting  models 
available  for  use.  The  models  are  generally  selected  based  upon 
the  type  of  decision  space  being  evaluated.  This  causes  the 
MCDM  methods  to  be  categorized  into  two  basic  types.  The 
first  deals  with  decision-making  given  a  continuous  decision 
space  such  as  mathematical  optimization  problems  to  find  the 
best  compromise  solution  amongst  multiple  or  infinite 
possibilities.  This  is  generally  known  as  Multi-Objective 
Decision  Making  (MODM).  For  the  most  part,  DoD  AO  As  will 
have  a  finite  or  discrete  number  of  alternative  solutions  to 
evaluate  and  therefore  use  the  category  known  as  Multi- 
Attribute  Decision  Making  (MADM)  [10]. 

There  are  numerous  deterministic  methods  for  Multi - 
Attribute  Decision  Making;  however  the  most  prevalent  in 
literature  seem  to  be  the  Weighted  Sum  Model  (WSM),  the 
Weighted  Product  Model  (WPM),  the  Analytical  Hierarchy 
Process  (AHP),  the  Elimination  and  Choice  Expressing  Reality 
(ELECTRE),  and  the  Technique  for  Order  Preference  by 
Similarity  to  Ideal  Solution  (TOPSIS).  Each  of  these  models 
have  proven  beneficial  in  their  own  right,  some  are  very  good 
when  the  decision  criteria  units  of  measure  are  the  same  (e.g., 
miles,  pounds,  seconds)  such  as  WSM,  WPM  and  AHP.  Others, 
such  as  ELECTRE  and  TOPSIS,  evaluate  each  criterion 
separately  using  weighted  normalization  and  pairwise 
comparisons  to  identify  the  best  alternatives  [11],  [12].  This 
research  has  selected  the  use  of  TOPSIS  due  to  its  employment 
of  an  ideal  point  method,  where  the  decision  alternative 
considered  most  suitable  is  calculated  based  upon  the  ranking 
of  summed  Euclidean  distances  from  both  the  best 
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and  worst  solution.  The  method  is  not  necessarily  complex. 

The  quantitative  information  provided  by  the  SMEs  for  each 
alternative  creates  the  decision  matrix  Z),  with  a  resultant  value 
at  the  intersections  given  as  (xij)  representing  the  performance 
measure  of  the  zth  alternative  in  terms  of  the  /h  criterion  as  seen 
in  Fig.  1. 

TOPSIS  then  normalizes  the  information  within  Matrix  D 
by  dividing  each  Xij  by  the  Square  Root  of  the  Sum  of  the  Square 
of  each  Xij.  This  results  comprise  a  new  matrix  N  of  normalized 
values  (nj)  using  (1). 

i 

rij  —  '  '  =  !’  2’  —>m,j  =  1,  2,  ... ,  n  (1) 

In  addition  to  the  SME  providing  their  expert  value  of  each 
criterion  on  the  alternative,  they  provide  a  weighted  importance 
to  differentiate  the  individual  criterion.  The  importance  weight 
assigned  to  each  of  the  criteria  must  equal  a  sum  of  1.  TOPSIS 
uses  the  importance  weighting  to  calculate  a  weighted 
normalized  decision  matrix  V  e.g.,  multiplying  the  normalized 
values  (ry)  by  the  criterion  weights  (wi,  W2,  W3  ...wn ;  where  X 
wn=  1.  This  matrix  is  depicted  in  Fig.  2. 

Next  the  matrix  is  evaluated  to  calculate  the  best  (Positive) 
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Fig.  2.  Weighted  matrix  V,  Evaluating  the  normalized  result  from  matrix  N 
by  multiply  each  result  by  each  importance  weighting. 

and  worst  (Negative)  possible  solutions  based  on  a  composite 
of  all  criteria  weighted  values.  The  best  and  worst  are 
determined  using  (2)  and  (3). 

A*  =  {(max  Vij  I  j  E  J),  (min  Vy  I  j  E  J)  1 1  -  1,2,3. .  .n}  (2) 

i 

=  {Vl*,  V2*,  V3*,  ...Vn*} 

A'  =  {(min  Vij  I  j  G  J),  (max  Vy  I  j  E  J)  1 1  =  1,2,3. . . n }  (3) 

i 

-  {V1-,  V2-,  V3-,  ...Vn-} 

Finally,  TOPSIS  evaluates  the  Euclidean  separation  distance 
of  each  alternative  to  the  best  and  worst  available  determining 
a  scored  ranking  for  the  outcome.  This  is  calculated  using  (4) 
and  (5).  This  is  depicted  in  Fig.  3  [13]. 


s*  =  V(X(  Vij-Vj.)2 

(5) 

s-  =  V(X(  Vij-Vj.)2 

(6) 

Fig.  1.  Decision  matrix  D  -  Evaluating  each  alternative  against  each  criterion. 
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Alternative 


Composite  of  the  worst  values 


Positive  Ideal  Solution 


Composite  of  the  Best 
Values 


Fig.  3.  Scores  for  alternatives  are  calculated  using  the  distances  from  the 
Positive  Ideal  Solution  (S*)  and  the  Negative  Ideal  Solution  (S'). 


C.  Importance  Weighting 

Once  the  criteria  were  agreed  upon,  the  SME  were  requested 
to  provide  importance  weighting  for  the  six  criteria.  Intuitively, 
the  weighting  responses  were  different  based  upon  the  opinions 
of  the  responding  COL  Program  managers  lean  toward  the 
importance  of  cost  and  technical  maturity.  Engineers  tend  to 
favor  technical  maturity  and  integration,  while  the  users  and 
resource  sponsors  place  significant  importance  on  schedule  and 
functional  capability.  Table  II  depicts  the  importance  weighting 
of  each  of  the  COI. 


III.  Example 

To  demonstrate  how  Multi-Criteria  Decision  Modeling 
might  benefit  a  decision  maker  in  the  selection  of  alternatives; 
an  evaluation  was  conducted  on  a  representative  example.  The 
scenario  for  the  example  was  simply  to  emulate  a  Marine  Corps 
acquisition  that  had  four  alternatives  from  which  to  choose  a 
Cyber-Security  system  closing  a  high  ranking  gap  from  the 
Capability  Based  Analysis.  While  this  example  uses  sample 
data,  the  exercise  successfully  demonstrates  the  impact  of  an 
expert’s  opinion  on  decision  criteria’s  value  and  importance 
weighting  with  respect  to  the  preferred  alternative. 

A.  Decision  Criteria 

The  first  step  in  the  assessment  was  to  identify  the  key 
decision  criteria  needed  to  compare  each  of  the  alternatives  to 
meet  the  stakeholder’s  goals.  It  is  important  to  ensure  the 
criteria  are  the  same  across  survey  population  and  therefore 
must  be  agreeable  to  all  of  the  SMEs.  In  addition,  the  criteria 
must  be  quantifiably  measurable  to  provide  numeric  values  for 
the  modeling  tool.  Some  of  the  offered  criteria  were  easily 
quantified,  for  example  a  vendor’s  offered  cost  or  scheduled 
delivery  time  is  unique  to  their  alternative.  Other  criteria 
however  required  a  numeric  scale  for  use  by  all  SMEs  such  as 
the  Defense  Department’s  Technical  Readiness  Level  scale  or 
a  value  given  to  the  total  Threshold  Requirements  versus  the 
number  the  vendor  states  their  alternative  will  deliver. 

B.  Communities  of  Interest 

We  next  determined  which  specialists  or  communities  of 
interest  (COI)  within  the  population  were  best  suited  to  evaluate 
the  alternatives  based  upon  the  criteria.  Diversity  in  evaluation 
expertise  provides  the  decision  makers  with  a  greater  range  of 
assessment  of  the  alternatives.  For  the  example,  our  criteria  and 
quantification  measures  were  distributed  to  a  selection  of  SMEs 
from  the  following  COL 

1 .  Program  Managers 

2.  Resource  Sponsors 

3.  Systems  Engineers 

4.  Users 

TABLE  I 

decision  Criteria 


Cost  of  Delivery  Technical  Functional  Integration  Contract 

Alternative  Schedule  Maturity  Maturity  With  Current  Availability 

Architecture 


TABLE  II 

IMPORTANCE  WEIGHTINGS  BY  COMMUNITIES  OF  INTEREST 


E>ecision  Weighted  values  weighted  values  Weighted  value^Weighted  values 

(-•n(ena  for  Program  for  Systems  for  Resource  for  Users 

Managers  Engineers  Sponsors  (Warfighters) 


Estimated  Cost  to  Procure,  Integrate  and  Deliver  Alternative 

25 

15 

10 

5 

Estimated  Time  Required  to  Procure,  Integrate  and  Deliver  Alternative 

i  u L 

15 

1  5 

20 

Estimated  Technical  Readiness  Level  of  Alternative 

20  p 

20 

45 

15 

Estimated  Percentage  of  Functional  Requirements  provided  by  Alternative 

17 

20 

25 

45 

Estimated  Level  of  Modification  Required  to  integrate  the  Alternative 

12  i 

10 

10 

10 

Estimated  ability  to  attain  the  Alternative  (Contract  Vehicle,  Legal  issues,  etc.) 

15  1 

20 

5 

5 

100 

100 

100 

100 

D.  Alternative  Evaluation 

Finally,  Each  SME  received  the  exact  same  descriptive  data 
of  the  four  alternatives  for  the  example  assessment.  The 
descriptive  data  was  created  as  realistic  as  possible,  however, 
with  reduced  complexity  to  simplify  the  value  determinations 
for  the  example.  Table  III  provides  the  data  set  used  to  describe 
the  alternatives. 


TABLE  III 

ALTERNATIVE  SPECIFICATIONS 


Alternatives  Estimated  Estimated  Estimated  Estimated  Estimated  Estimated 
Cost  for  Completion  Technology'  Functionality'  Integration  Programmatic 

Alternative  Schedule  Readiness  Capability  Readiness  Accessibility 


(Months)  Level  Plug N Play... 7-10  Contract Avail..7-10 


To  complete  the  analysis,  each  alternative  was  ranked  by  the 
SMEs  from  the  four  individual  COI  and  the  values  were  entered 
into  the  MCDM  tool  TOPSIS  for  evaluation.  To  ensure 
differentiation  between  the  COI,  each  was  modeled 
individually  maintaining  the  importance  weighting  for  their 
community. 


IV.  Evaluation  results 

The  results  of  the  four  independent  TOPSIS  simulations 
based  on  the  COI  responses  provided  the  decision  maker  with 
three  separate  recommendations.  The  PM  and  User 
communities  both  selected  Alternative  number  4,  while  the 
Systems  Engineer  and  Resource  Sponsor  communities  selected 
Alternatives  1  and  2  respectively.  This  can  be  seen  in  (4).  The 
generally  accepted  method  to  select  a  best  value  alternative 
using  the  four  TOPSIS  simulations  is  to  cumulatively  sum  the 
scores  of  the  four  COI  using  (7)  and  evaluate  the  results.  As 
depicted  in  (5),  our  example  analysis  clearly  shows  Alternative 
4  as  the  winner  with  35%  of  the  scoring  value. 


With  a  little  discussion,  each  of  the  SME  agreed  on  the 
example’s  decision  criteria  as  depicted  in  Table  I. 
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Selection  Scores  for  PM 
Weighting  Using  All  Criteria 


Si  0.5530  ■ 

a  b 


Alternative  Alternative  Alternative  Alternative 
12  3  4 


Selection  Scores  for  SE 
Weighting  Using  All  Criteria 


^0^85^8 

9  H 


Alternative  Alternative  Alternative  Alternative 
1  2  3  4 


Selection  Scores  for  RS 
Weighting  Using  All  Criteria 


Selection  Scores  for  User 

Weighting  Using  All  Criteria 


Alternative  Alternative  Alternative  Alternative 
1  2  3  4 


Alternative  Alternative  Alternative  Alternative 
1  2  3  4 


Fig.4.  Alternative  selections  using  individual  COI  weightings 


V.  Sensitivity  Analysis 

Conventional  wisdom  would  suggest  that  this  cumulative 
method  would  mediate  any  bias  or  subjectivity  on  the  decision 
criteria  input  amongst  the  different  COI.  It  also  seems  to  justify 
the  benefit  of  diversity  in  evaluation  expertise.  For  example,  if 
the  decision  were  left  to  the  Systems  Engineers, 

Alternative  1  would  have  been  recommended.  However,  our 


ALT.  1 
ALT.  2 
ALT.  3 
ALT.  4  p 

0.0000 

Fig.  5.  Cumulative  selection  based  on  individual  COI  weightings 

research  is  to  investigate  how  sensitive  the  input  may  be  to  these 
biases  and  if  the  weightings  of  the  individual  criteria  were 
actually  causing  uncertainty  in  the  decision  results.  What  if  two 
or  more  of  the  criteria  were  to  either  combine  to  arbitrarily 
increase  the  likelihood  of  a  preferred  alternative?  Or  similarly, 
if  two  or  more  criteria  offset  each  other  to  decrease  a  preferred 
alternative,  would  this  change  the  results?  It  seems  logical  to 
hypothesize  that  relationships  between  the  individual  criteria 
could  subjectively  affect  the  outcome  enough  to  cause  a  rank 
reversal  in  the  recommended  solution. 


TABLE  IV 

SENSITIVITY  ANALYSIS  WEIGHTING  NORMALIZATION 


Cost 

Schedule 

Technical 

Function 

Integ 

Contracts 

Cost 

0.00 

0.27 

0.29 

0.28 

0.27 

0.28 

Schedule 

0.16 

0.00 

0.15 

0.14 

0.13 

0.14 

Technical  Maturity 

0.25 

0.22 

ooo 

0.23 

0.22 

0.23 

Functional  Maturity 

0.22 

o.id 

0.21 

0.00 

0.19 

0.20 

Integration 

0.17 

0.14 

0.16 

0.15 

0.00 

0.15 

Access  to  Contracts 

0.20 

0.17 

0.19 

0.18 

0.17 

0.00 

Sum  of  Weights 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

To  test  this  hypothesis,  an  additional  TOPSIS  simulation  was 
conducted  using  the  same  alternatives,  COI  and  weightings  but 
insolating  each  of  the  decision  criteria  independently.  This 
assessment  was  to  identify  which  factors  in  the  model 
contribute  most  to  the  variance  in  the  conclusions.  This 
evaluation  is  broadly  defined  as  Sensitivity  Analysis  (SA)  [15]. 


Selection  Scores  for  PM  Weighting 

after  Sensitivity  Analysis 


Alternative  Alternative  Alternative  Alternative 

- 1 - 2 - 3 - 4 - 


Selection  Scores  for  SE  Weighting 

after  Sensitivity  Analysis 


Alternative  Alternative  Alternative  Alternative 
12  3  4 


Selection  Scores  for  RS  Weighting 

after  Sensitivity  Analysis 


Selection  Scores  for  User  Weighting 

after  Sensitivity  Analysis 


FI 


Alternative  Alternative  Alternative  Alternative 
12  3  4 


Alternative  Alternative  Alternative  Alternative 
12  3  4 


Fig.  6.  Recommended  alternatives  based  on  The  Sensitivity  Analysis 


The  model  isolated  the  interaction  between  criteria  by 
removing  each  individually  and  distributing  its  original  value 
proportionately  across  the  remaining  five  criteria.  This  same 
method  was  conducted  for  each  of  the  six  criteria  as  depicted  in 
Table  IV.  The  TOPSIS  model  was  conducted  for  each  COI 
importance  weighting  rendering  solution  recommendation  for 
each  run.  (SpmI,  Spm2...Spm  6,  SseI,  Sse2...Sse6  and  so  on.) 
The  selection  scores  for  each  COI  were  then  calculated  once 
again  using  (7). 


Fig.7.  Recommended  best  value  solution  based  on  cumulative  Sensitivity 
analysis  scores. 


The  results  of  the  twenty  four  independent  TOPSIS  model 
runs  using  the  sensitivity  analysis  responses  interestingly  shows 
a  dramatic  increase  in  the  selection  for  Alternative  3.  The 
change  was  so  dramatic  that  it  rendered  the  distinction  between 
Alternative  3  and  4  negligible.  This  is  depicted  in  Fig.  6  and  7. 


VI.  Future  work 

The  example  problem  presented  in  this  article  describes  an 
issue  that  suggests  more  research  is  required  to  identify  the 
effects  of  subjectivity  and  bias  on  the  input  to  MCDM  tools. 
Our  research  will  be  conducted  over  the  next  year  as  the  subject 
of  dissertation  work  for  George  Washington  University.  Our 
objective  is  to  coordinate  with  the  United  States  Marine  Corps 
to  evaluate  an  actual  Cyber  Security  acquisition  decision 
duplicating  the  example  methodology  described.  With  the 
actual  information  as  a  basis,  a  comparative  study  can  be 
focused  on  the  use  of  these  techniques  to  identify  how  sensitive 
the  input  criteria  to  MCDM  tools  are  and  to  identify  techniques 
to  reduce  the  uncertainty  for  evaluations. 


Our  methodology  is  to  utilize  sensitivity  analysis  techniques 
such  as  regression  analysis,  correlation,  and  multicollinearity 
assessment  to  identify  the  change  and  error  on  the  input  to  the 
model.  Regression  analysis  evaluates  if  the  probability 
distribution  of  a  random  variable  y  may  depend  on  the  value  of 
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some  predictor  variable  x.  The  regression  analysis  sets  up  the 
evaluation  of  this  relationship  using  a  single  predictor  variable 
(simple  linear  regression)  or  multiple  predictor  variables 
(multiple  linear  regressions)  [16].  Correlation  measures  the 
strength  or  linear  association  of  the  two  variables  i.e.,  v  and  y. 
If  there  is  a  strong  correlation  between  the  two,  v  may  better 
predict  the  future  value  of  y,  using  (8)  [17]. 

y  =  P  q  +  x  (8) 

As  an  example,  an  initial  analysis  of  correlation  amongst  the 
decision  criteria  in  the  example  problem  suggests  a  strong  linear 
relationship  between  cost  and  each  of  the  other  criteria.  If  there 
happens  to  be  a  high  correlation  such  as  this  between  many  of 
the  independent  or  predictor  variables,  the  issue  of 
multicollinearity  may  arise  [18]. 

VII.  Summary 

Accurate  decisions  concerning  which  alternative  solution  to 
select  early  in  an  acquisition  can  have  dramatic  impact  of  the 
ultimate  program  cost  and  schedule  of  a  Cyber  Security 
Solution  or  the  Cyber  system  of  systems.  Multi-criteria 
Decision  Analysis  techniques  have  been  demonstrated  as  useful 
tools  to  support  decision  making  in  the  DoD  acquisition 
community.  Studying  the  relationships  between  the  separate 
inputs  through  sensitivity  analysis  is  expected  to  reduce  bias  on 
the  input  criteria  and  increase  the  accuracy  of  the  ultimate 
decision. 
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